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The techniques and accompanying problems of 
on interventions are explored* As a result of the 
previous studies (Colorado and Kentucky Ethnographic 
and Texas, 1981) a hybrid-method of documenting 



ABSTRACT 

collecting data 
analyses of two 
Studies, 1979; ^ 

interventions was developed for use in the Principal-Teacher 
Interaction Study (PTI). This paper traces the development of the 
methodology for documenting interventions used in the PTI. Nine 
elementary school principals (three each from Colorado, Florida and 
California) served as the primary subjects and informants on 
intervention in this study. The study sites varied by principal's 
concerns, year of implementation, and innovation being implemented, 
as well as by school district. Procedures included: bi-weekly, 
\telephone calls, interviews, logging, document collections and field 
\>otes by researchers; and multiple perspectives and sources* of data. 
In all three studies, the informants provided data on the 
interventions in which they themselves, as well as others or events, 
were the sources of action. The methodology employed for the PTI 
study is as effective and efficient a self -report ing technique in 
developing the total picture as are the more expensive ethnographic 
methodologies . (PN) 



************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



ERLC 



, ' U.S. DEPARTMENT OF EDUCATION 

} ^ NATIONAL INSTITUTE OF EDUCATION 

EDUCATIONAL RESOURCES INFORMATION 

CENTER (ERIO 
5^ Ti»t$ docomeni has beea repfoducwl as 
recv'ived <^om the person or of9>ni/ai«n 

M.r.Of cJwnge* havo been «vjde 10 impfOve 
- reproduciioo <iyo^«ty 

♦ Points, of vM?w or 0P»nK>nSswled in this docu 
mtfni do not nece^afjJy icpfwent oH»cta1 NiE 



po$,iiK>n pr policy 



UJ 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



.METHODS FOR DOCUMENTING INTERVENTIONS: 
STRENGTHS OF A HYBRID MULTI- INFORMANT APPROACH 



Marc i a Goldstein 

William Rutherford 

Research and Development Center for Teacher Education 
The University of Texas at Austin 



NO 
si 



N Paper presented at the annual meeting of the 

^' American Educational Research Association 

New York City, 1982 



o - 2 



ERIC 



METHODS FOR DOCUMENTING INTERVENTIONS: STRENGTHS OF 

A HYBRID/MULTI-INFORMANT APPROACH^ '^'^ 
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For the past five years the CBAM Project at the Research and Development 
Center for Teacher Education at The University of Texas, Austin, has been 
developing the concept of "interventions" as it applies to the facilitation of 

school change. Three studies have oeen conducted in an attempt to further 

> 

understanding of this phenomenon and to subsequently develop a theory of 
intervening. To understand the reasons for and the significance of this 
research on interventions, it is useful to review a bit of educational 
history. ' ■ 



Background 

National concern about educational effectiveness surged upward in the 
late 1950' s, after Sputnik. Beginning in the 1960's the primary approach to 
responding to this concern was the "new program" method— develop new programs. 



This paper was presented at the annual meeting of the American 
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put them into the schools, and supposedly education will be improved. 
Millions of dollars were spent by the government and commercial interests in 
the development, dissemination, and support of literally thousands of these 
new programs V- 

However, in the 1970' s, evidence began to indicate that in spite of the 
millions of dollars expended, the "new programs" approach often was not 
producing the hoped-for outcomes. At this same time, members of the CBAM 
project staff at the Texas Research and Development CenteV for Teacher 
Education initiated research efforts which resulted in the understanding that 
the key to school improvement is not "new programs," but the individuals 
within the school: teachers rnd principals (George & Rutherford, 1977). It 
is therefore important to know how individual teachers use and feel about new 
programs or innovations in order to understand how schools change. The CBAM 
project developed two concepts and assessment techniques — Stages of Concern 
(Hall, George & Rutherford, 1979) and Levels of Use (Hall, Loucks, Rutherford 
& Newlove, 1975) — which address the question of wljf^^ teachers do and are 
concerned about when implementing a new program. ^ 

Later, the CBAM project began to focus on those individuals who have 
responsibility for initiating and supporting school change efforts, the 
"change facilitators." Just as it is important to understand what teachers 
do, it is equally important to understand what* those who sponsor improvement 
efforts do to facilitate change. This led to the study of "interventions," 
•defined as those actions or events that influence use of an innovation (Hall, 
Zig'armi & Hord, 1979). Early studies on interventions done by the. CBAM 
research staff (Hall, 1979; Hair, Hord & Griffin, 1980) and research conducted 
by others (Little, 1981; Stal lings, 1981) revealed that the key change 
facilitator in school improvement efforts* is the school principal. Current 
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CBAM research, The Principal-Teacher Interaction (PTI) Study, now in the data 
analysis stage, focuses on interventions made by this key individual. 

The study of interventions poses difficult questions and problems for the 
researcher. First, the researcher is faced with the methodological challenge 
of devising procedures for collecting and documenting interventions. This is 
followed by the equally challenging task of analyzing the intervention data. 
Using th^ knowledge and experience gained from previous research, the CBAM 
^ project staff has^developed both a suitable methodology for documelftirjg 
interventions, as well as a framework for approaching analysis, the Taxonomy 
of Interventions (Hall Zigarmi & Hord, 1979). 

This paper traces the development of thq methodology for documenting 
interventions used in the PTI Study. Two previous studies conducted which 

c 

contributed to development of the documentation methodology are discussed. 

"J * 

The paper attempts to show how refinement of the concept of interventions and 
the methods used to research the concept went hand-in-hand. The strengths and 
limitation's of the resultant hybrid/multi-informant methodology are discussed 
as are implications for*use in future research done both within and outside 
the realm of school change. 
■* 

Guiding Principles from the Literature 

No attempt is made here to provide an extensive review of the literature 
on the merits of quantitative and qualitative methodologies in research. 
Rather, what will be presented is literature that has direct application for 
our research techniques. We do not feel compelled to engage in the 
controversy over the merits of qualitative versus quantitative methodologies. 
Although the issue has certainly -not been settled, the battlelines are 
becoming increasingly obscure, and the flag of truce has been raised for those 
who cannot, or do not wish to choose b^ween the rpethods. Patton states: 



The issue of selecting methods is no longer one of the dominant . 
paradigm versus the alternative paradigm, of experimental designs 
with- quantitative measurement versus holistic-inductive designs 
based on qualitative measurement. The debate and competition 
between paradigms is being replaced by a new paradigm-- a paradigm 
of choices* The paradigm of choices recognizes that different 
methods are appropriate for dif;ferent situations (1980, p. 19-20)* 

Reinhardt and Cook take the position that . • qualitative methods need not 

only to be used to discover which questions are interesting to ask and 

quantitative procedures need not only be used to answer them. Rather, each 

procedure can serve each function." (1979, p. 13-14). 

Having literature sanction to use one or both methodologies was 
comforting, but not sufficient to guide our research. Wax (1971) helped to 
clarify our position in her explanation of four types of researchers; she 
described one group this way: "They may seek an inside view, but also enjoy 
working with models. Or they may see themselves as model builders, but enjoy 
stepping inside a culture or a society in an attempt to understand what the 
models mean to the people who use them." (p. 45). Through the development of 
the Taxonomy of Interventions, a tentative model for analyzing interventions 
had already been (jQveloped,.. The major task in the PTI study wa§ to get an 
inside view of school principals and the interventions they make, to convert 
these data into a form suitable for verification against the taxonomy, and to^ 
assess the taxonomy against the data. 

There v/as a need to have not only an inside view of the principal and the 
school, but to have an accurate, reliable view. It was certainly not feasible 
to have a researcher permanently located in the study schools, so how could we 
otherv/ise be assured of an accurate and reliable picture? In his discussion 
of analyses of social settings, Lofland (1971) offered some useful guidelines 
for research such as that conducted in^the PTI study. He pointed out that in 
our complex society it is less likely that we will know a people and more 
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likely that we will only know about a people^ To compensate for this 
difference between knowing and knowing abo.ut, he suggests a reasonable 
substitute for ''face-to-face knoWi-ng" (p* 3), and characteristics of that 
substitute* Those who would report on a people should first of all be close 
to them, meaning both in a social sense and in physical proximity* This 
.closeness should extend* over a significant period. Reports flowing from this*" 
research relationship should give attention to minute matters and should be 
truthful and factual* These reports should give • pur^e description of 
action, people, activities, and the like" (p* 4)* Finally, direct quotations 
from informants should" be used. 

Yin (1981) offered some very helpful guidelines for collecting and 

5 

analyzing case studies. He suggested that sources of information for case 
studies include face-to-face interviews, telephone interviews, records, 
documents, and memoranda, illustrative materials (e.g., newsletters), and 
on-site observations. To heli3 prganize information from all these sources, 
Yin recommended use of a protocol to identify the iff?ormation desired and the 
sources from which the information would be gleaned. Furthermore, he 
recommended that the major informants review the factual portions of case . 
studies after the data are collected. Quantitative tabulations could be 
applied to data collected in case studies when there are enough studies that 
could be synthesized across one or a few critical variables. In 'effect, this 
constitutes a survey of the case studies (Yin and Heald, 1975; Yin, et al . , 
1976) and this survey could be subjected to statistical analysis. 

The intervention studies done by the CBAM project reflect a blend of 
qualitative and quantitative considerations as was suggested by Yin, Wax, 
Lofland and others. ) 
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Three Studies Aimed at Documenting and Understanding Interventioris 

The CBAM Project has conducted three studies in which interventions were 
identified and documented* These studies demonstrate the developmental 
progression of the project's conceptualization of interventions as well as the 
refinement \n methods of documenting and analyzing interventions. All studies 
employed qualitative "descriptive but not yet quantified (Meyers, 1981, p. 
161)" procedures for collecting intervention data. However, documenting 
procedures became somewhat more structured as the defining characteristics of 
interventions evolved. 

Study 1 (A and B): Colorado/Kentucky Ethnographic Study 

The first study to be discussed is actually two studies which were 
-conducted concurrently at two different sites, but which encompassed many of 
the same aims, purposes and methodologies. In both studies, .the'aim was to 
collect descriptions of actions or events which might potentially affect 
implementation of the innovation being studied. In one of the studies (lA), 
attention was focused at the district level, and in the other (IB) the focus 
was on an individual school. Ethnographers were employed at each of the two 
sites. The ethnographers were given the project's working definition of an 
. intervention to guide their data col lection, but were given very little 
structure otherwise. 

The first of the two studies took place in a large suburban school 
district in Colorado and focused on the implementation of a district-wide 
revision of a science program for grades 3-6. One aspect of this effort 
involved the documentation of what was done by district staff developers and 
the science department staff to facilitate implementation of the revised 
science program. 

In order to accomplish this task, the following procedures were employed:. 
» 6 ^ ^ . 
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1. A full-time ethnographer, located on-site at the district office, was 
employed during the first, two years of the implementation effort. This 
individual described the activities of district administrators, staff 
development and science department staff, as well -as science teachers and 
other teachers who served as change facilitators in the implementation effort. 
The ethnographer's field notes for the^two years resulted in 41 volumes 
(approximately 3,000 pages) of intervention aRd contextual descriptions. 

2. Several of the district staff members kept records on their 
activities as participant- observers. 

3- The CBAM Project staff kept notes of their own activities which had 
potential for affecting implementation when they were on-site for purposes of 
planning and quantitative data collection (another aspect of the study). 

4. District memoranda and other documents related to the science program 
implementation, including newspaper reports, were also collected to add to 
this qualitative data base. 

Simultaneously with this stlidy of a science program implementation in 

'\ 

Colorado, the CBAM project also monitored the implementation of Reality 
Therapy as a new approach to discipline at a junior high school in Kentucky 
(Zigarmi, 1979). This innovation was one part of a Teacher Corps project 
de>?eloped and implemented through the collaboration between the junior high 
school and a nearby university. The main focus of this effort was to study 
actions taken by school, school district and university personnel to . 
facilitate the implementation of the new discipl*ine strategy. 

An ethnographer was employed to document actions or events occurring 
which might have some influence on implementation. The ethnographer was 
on-site for two days a week during the first year of implementation and two to 
three days a months* during the second yean. The ethnographer observed 



university change facilitators meetings with the. principal , teaching teams and 

individuals, the principal's meetings with teachers, as w^n as workshops and 

weekly faculty meetfngs. She also conducted frequent informal interviews -with 

change facilitators and project staff, universiiiy faculty members, the 

principal, the team leaders and individual teachers. yi]en away from the site, 

the ethnographer maintained contaqt with a few significant individuals by 

phone. She also had access to meeting minutes, key district documents like 

school. poli-cies, and media reports. All of these information sources resulted 

in 30 volumes (approximately 2,000 pages) of descriptive narrative on the 

two-year implementation effort. • \ 

\ \ 
During the two years that studies lA and B were* proceeding^ the CBAM 

t 

projiect conducted monthly meetings of the staff, including both ethnographers 
at the project office- in Texas for the purpose of clarifying the concept of 
interventions and developing a "taxonomy" of interventions. Work towards this 
goal continued for another year following completion of the field work and 
resulted in the development of two frameworks for analyzing interventions: , 
Levels of Interventions which describes a hierarchy of intervention types from 
incident to tactics and strategies, up to policy interventions (Hall, 
Zigarmi, & Hor^l, 1979) and a coding schema for specifying the subparts of 
interventions, such as the source, target and function (Hord, Hall & Zigarmi, 
1980). 

The Colorado/Kentucky study not only contributed to our understanding of 
the concept of interventions and ethnographic methodology, 'but also mcide 
apparent the importance of the role of the principal in school change efforts. 
Current literature on the role of the principal (e.g.. Little, 1981; 
Stallings, 1981), also supports this view. A decision was made to focus 
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future research by the CBAM project upon interventions made by the school 
principal to facilitate t^e implementation of innovations. 

Study 2; The Pilot Study - ' » " - 

In preparation for' the major research study on the rjole of the principal 
in school change, the CBAM Project conducted a 3-month pilot study in ten * 
schools in Central Texas (Griffin, Goldstein & Hall, 1981). The main purpose 
of this pilot study .was to investigate different ways .of collecting 
information about interventions made by school principals as they are 
facilitating implementation of innovatton^ in their schools. 

Since the project had ga^ined some experience with the use of ethnographic 

techniques in earlier studies^ and sjnce the^ conceptualization of 

interventions had evolved considerably, it was decided that various forms of 

. " \ 

structured self-report would be tested in this \>Mot. The ten schools that 

participated in the pilot were involved in the implementation of different 

innovations and were at different phases of the implementation process. Two 

principals were assigned to each of five self-feport procedures. 

! 

Before the study began, members of the CBAM research team conducted 
individual sessions with each of the principals to acquaint them with the 
project's .definition and conceptualization of interventions and to train them 
to identify aiid describe interventions me^de by^ themselves and others. 
Regardless of the procedure group to which thty were assigned, each principal 

...was trained and asked to describe interven1:ions in terms of : 

^ • ^ / . 

1. What the action was 

<« 

2. Wh^ the action was taken^ 

3. To whom the action was targeted 

i V 

4. When the action occurred 

5. How long it took to complete, dnd 



6- What the perceived effects of the action were- 

Jhe five variations of self-reporting procedures tested were as follows. 
L A written log maintained on" a weekly basis by the principal . 
'Principals described each of their* own and oth?ft facilitators' interventions 



by recording them on logging sheets (Figure 1), including information on the 
stx ^"categories mentioned above. Principals were asked to faa.il their logs to 
the^^'project office ^each week. * 

\\ 2: A weekly report sljbmitted via audio tape , Jhe procedure for thi5 

\ \ ' ' 

method. was basically the sam^ as for the written log, except that principals 



in this group tape-recorded their information in place of writing it down. ^ 
. They 'used vthe same six informational categories as did those completing the 
written log^^ They were asked to mail* their ^ tapes -to the project office 
weekly. A* project -staff men\ber later filled out log 'sheets based on the 
informatiori 0^ the, tapes* , 

3. Weekly reportlj^rovided via telephone . Project members .contacted 
.princigals by telephone once a ^ wfeek in order to obtain descriptions of ^ 
interventions, made b^ the principal and .others the .previous week. The project 
member probed for more information about actions mentionefl by t\\e principal, 
when necessary^ to obtain complete data. These conversations were, 
tape-recorded and later transferred to logging sheets the project staff 
member. . , ' ' 

4 & 5. Face-to-face' interviews with principals— two variations . Once a 
m^th a project staff member visited the school and interviewed the principal 
about interventions that had occurred during the- last month. Two forms of 
str/ictured interviews were used--one based on the concept of game plan 
components, which is one of the Levels of Interventions ^Hall , Zigarmi & Hord, 



PRINCIPAL LOG 



Coding 
for 

R&D Date 
Use 



Figure 1 



Name 



Il)P 2 



Week of 



, 19 



Target (s) 



Actions Taken: 
What Did You Do? 



Intent: 
^^y Did You Do It? 



How Long 
Did Your 
Action . 
Take? 



What were 
the 
Effects? 



: Q ir log(s) each week to the UTR&D Center in the envelopes we have provided. 



1979), and one based on the concept of Stages of Concern (Hall, George & 
Rutherford, 1979). The interviewer probed to obtain complete information on 
the six components for each intervention mentioned. The interviews were 
recorded and project staff members later ti*ansferred the intervention data to 
logging sheets. 

At tlie conclusion of the pilot study, three teachers from 5 of the sample 
schools (15 teachers total) were interviewed to determine whether they could 
recall and verify the interventions reported by the principals. These' 
teachers were asked a general question about actions or occurrences which 
might have influenced their use of the innovation and then they were asked 
questions directed at determinirig if they could verify specific interventions 
which had been reported by their principal. 

Finally, at the study's conclusion, each principal was interviewed by a 
project staff member to determine his/her reaction to participation in the 
study. In addition to general reaction^ to the study, the debriefing included 
questions about reactions to the sp£cific procedures tested, problems caused 
by the procedures, and whether or not and how involvement in the study 
affected actions taken with respect to the innnoVation. 

The pilot study data were analyzed to determine the effectiveness and 
efficiency of the various documentation methods in order to select the 
procedure to be used in the major study on principal interventions (the 
Principal-Teacher Interaction Study). Several aspects of effectiveness and 
efficiency were considered in the analysis: level of compliance of principals 
v/ith the procedures; sufficiency, completeness and appropriateness of the 
data; whether the method yielded "bonus" (e.g., contextual) data; agreement 



between the principal's self-report and the teacher's verifications; the 
satisfaction and comfort of both researchers and subjects with the collection 
-procedures and, finally, cost-efficiency (results are reported in Griffin, 
Goldstein & Hall, 1981). As a result of these analyses, a hybrid-method of 
documenting interventions was developed for use' in the Principal -Teacher 
Interaction Study. 

study 3: The Principal-Teacher Interaction Study (PTI) 

The Principal-Teacher Interaction' Study focused specifically upon school 
principals to determine what they do to aid the implementation of new programs 
in their schools. Three major questions guided this study: (1) What do 
pTtTrcipFl3^D"ffS-ctTOTg^ri^ tatars?^ , (2 ) How-da the- corrcern^ of i)ri nci paH- 
effect their functioning as change facilitators? and (3) What is the 
relationship between administrator concerns, the interventions they make and 
their Effects on teachers? 

Nine elementary school principals served as the primary subjects and 
informants on interventions in this study. Three principals were selected 
from school districts in Colorado, Florida and California. The principals 
were chosen by district administrators to represent change facilitators with 

varying kinds of concerns— impact concerns, task concerns and personal 

concerns. Each of the three school districts were at different points in the 
process of implementing a district-wide innovation. The California school 
district was beginning its first year of implementation of a new composition 
writing program; the Florida district was beginning its second year of 
implementation of a unified math curriculum; and the Colorado district was 
into its third year of implementation of a revised, science curriculum. So, 
the study sites varied by principal's concerns, year of implementation, 
innovation b&ing implemented, as well as by school distFict. 
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As in the pilot study, principals were trained on an individual basis to 
identify and describe interventions made by themselves and others. By this 
point in time, the CBAM project's conceptualization of interventions was quite 
well defined. As mentioned previously, concept development culminated in a 
taxonomy and framework which enabled the researcher to code individual 
interventions on a number of dimensions (Hord, Hall, & Zigarmi, 1980): level, 
sublevel, source, target, function, medium, flow and location. In these 
sessions, principals were trained to provide sufficient information in 
reporting interventions to allow for coding on these dimensions. At the end 
of the training sessions, the principals were left with training notebooks 
which contained intervention and dimension .definitions and examples. They 
were also provided with specially-^'asigned calendars which had notations on 
important dates in the study and ample spaqe' for intervention notes. 

Study Procedures 

The procedures developed for documenting interventions in the PTI 
study combined the most successful elements of procedures tried in the pilot 
and in earlier research. The procedures included bi-weekly telephone calls, 
on-site interviews, informal logging, document collection and field 
note-taking by researchers. The study also included multiple perspectives and 
sources of data. Though the principal was the focus of the study, others, 
such as teachers, assistant principals and district resource personnel, also 
served as informants on interventions, A summary and time-line of these 
procedures, which are described in more detail below, is provided in Figure 2, 

Members of the CBAM research team contacted study principals (and 
assistant principals^ by telephone on a bi-weekly basis during the 1980-81 
school year to collect data on interventions related to the innovations beina 
studied. The researcher began each call by asking the principal what actions 
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he/she had taken with respect to the innovation during the last two weeks. 
Then the principal was asked about other actions or events that had occurred 
which might have some effect on the implementation. A? the principal reported 
each intervention, the interviewers would probe for all the information 
necessary for coding and for understanding the action and its effects. To 
facilitate this process, principals were asked to keep notes on the calendars 
provided by the project cluring the two week interval between phone calls. 

Face-to-face interviews were conducted with principals (and 
assistant principals) on-site three or four times during the study period. 
These interviews focused on obtaining higher level interventions, like game 
plans or strategies, as well as background, philosophy and contextual 
information. Teachers also were interviewed during these ^on-site vis'its. 
They were asked open-ended questions about interventions. .v/hich had occurred, 
as well as specific questions pertaining to interventions which had been 
'reported by the principal (or assistant principal). 

District level resource personnel who acted as change facilitators 
for 'the innovation served as additional informants on interventions. They 
were questioned about their own interventions at the study site schools, a^s 
well as interventions that they knew others had performed in these schools. 
Interviews were conducted by telephone or on-site on a semi-regular basis 
--from once a month to once every three months, These interviews proceeded in 
much the same way as the principal interviews. 

CBAM project staff members also served as sources of intervention 

data. The staff members noted interventions they themselves had made, as well 

c 

as interventions they had observed while on site at the schools. 
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Figure- 2 

PTI Data Collection Procedures and Timeline 



Principals . May 1980 

(and Assistant Principals) 



Oct. 1980 Jan: 1981 



May 1981 



Intervention phone 
interviews 

Intervention face-to-face 
interviews 

Field note-taking 

Context survey 

Change facilitator 
Stages of Concern 
Questionnaire 



bi-weekly 



X 
X 
X 



Teachers 

Intervention face-to- 
face interviews 

Stages of Concern 
questionnaire 

Levels of Use interview 

Innovation Configuration 
interview 

School Climate Survey 



X 

X 



X 
X 



X 
X 



District Personnel 

Intervention phone 
interviews 

Intervention face-to-face 
interviews 

Context Survey 



. intermittent 



X 
X 
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Data on a number of variables other than interventions were also 
collected as part of the study: 

1. Variables which were used as indicators of the effects of 
interventions, on teachers — teachers' concerns about and use of the 
innovation, measured by the Stages of Concern |ilestionnaire (Hall* George & 
Rutherford, 1979) and the Levels of Use (Loucks, .Newlove & Hall, 1976) and 
Configurations Interview' (Heck, Stiegelbauer, Hall & Loucks, 1981). 

2. Principal (and assistant principal) concerns about facilitating 
the implementation of the innovation, measured by the Change Facilitator 
Stages of Concern Questionnaire (Rutherford, Hall & George, 1982). 

3. School and district contextual variables. The work of James and 
Jones (1974, 1979) influenced the CBAM project's conceptualization of this set 
of variables. Project members collected information on the more objective- 
type characteristics of the school and districts, as weU_as data on school 
climate as perceived by teachers (Hall & Griffin, 1982). ^ 

In addition to the structured data collection procedures, research 
staff kept ethnographic notes on their visits to the schools, as well as notes 
on their impressions, hunches .and possible hypotheses about principal 
behavior. The project staff also kept up with school district news and 
subscribed to local newspapers in each district to monitor any external events 
that might affect innovation implementation. ^ , 

Summary of Methods of Documenting Interventions 

Figure 3 organizes documenting methods employed in the three studies 
discussed above according to two dimensions: (1) the informants or sources 
supplying information about interventions, and (2) the procedures by which the 
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informants provided the data to the research project. In all three studies, 
the informants provided data on interventions in which they themselves were 
the source of the action^ as well as interventions in which others or events 
were the sources. In Studies 2 and 3, a large proportion of the interventions 
had the principals as source because the principal's behavior was the focus of 
the study and because the principal served as the primary informant in these 
studies. In these studies, the other informants — teachers, external change 
facilitators, and the researchers — reported on the principal's actions as 
wen as actions they themselves and others had taken which might have affected 
implemenfation of the targeted innovations. As far as the procedures were 
concerned, while observation and document review played a role in the studies, 
the majority of procedures employed and tested were different forms of 
self-report. 

Another dimension which varied across tf^e studies, and which might have 
been added to Figure 2, is the intensity or frequency of the data collection 
methods. In Studies lA and IB the data collection was continuous. or close to 
continuous, involving a full-time ethnographer at one research site (Study lA) 
and a part-time ethnographer at the other (Study IB). In Studies 2 and 3, 
documentation procedures were more intermittent: the telephone procedures 
involved contact on a weekly or bi-weekly basis and the on-site interviews ' 
'i>(0lved contact 3 or 4 times during the study period. Frequency of contact 
is emphasized because it is one variable which was found to be an important 
determinanKqf a documenting procedure's effectivenessv. 



Figure 3 



A Summary Table of Methods of 
Documenting Interventions 



Data Collection Procedures 
Oral Reports 



Informants Observation C Face-to-Face Telephone Tape^ Written Reports 



Principal/AP 




Study 1 
Study 2 
Study 3 


Study 1- 
Study 2 
' Study 3 


Study 2 


4 

•Study 2 
Study 3 
(minimal ) 


Teachers 




btuay i 
Study 2 
■Study 3 








External 
Change 

Facilitators 




Study 1 
Study 3 


Study 3 




Study 1 


Outsiders 
(District 
Adm. , news 
media) 




Study 1 
(minimal ) 






Study 1 
Study 3 


Researchers 


Study 1 
Study 2 
(minimal ) 
^ Study 3 
(minimal ) 











Study 1: Colorado/Kentucky Ethnographic Study 
Study 2: Pilot Study 

Study 3: Principal -Teacher Interaction Study , 

NOTE: each informant provided data on themselves and on all other informants. 
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strengths and Weaknesses of the Various Procedures for Documenting 
Interventions 

The following discussion of the strengths and weaknesses of procedures 
used by the CBAM project for documenting interventions will be based upon data 
collected in the pilot study (Study 2), debrief ings with principals done at 
the conclusion ol Studies 2 and 3, and the personal impressions of the 
research team (e.g., Hall, 1980; Kord, 1980). Judgements of strength and- 
weakness will focus on a number of criteria that have been deemed important to 
the study of interventions as well as. to research, in general: 
cooperation/compliance with the research effort, codability and completeness 
of the data set, representativeness and range of interventions, expense and 
ease of the procedures, degree to which the. procedures intervenje upon behavior 
cf participants or interrupt normal routine, degree to iWhich they offer an 
understanding of the total context, and amount of data reduction required for 
analysis. 

Ethnographic Observation ^ 

When observation is a continuous or semi-continuous data collection 

activity, it has the unique advantage of providing the most complete 

information on interventions. The researcher has the opportunity to record 

small, seemingly insignificant incidents that might have gone unmentioned in 

an interview or written log. The researcher can record elements of the action 

or event that are necessary for coding, as wetl as descriptive notes to 

explain the conte'Jct and links between interventions. This offers the 

potential of grasping the dynamics of the whole Implementation effort. 

Observation also avoids the pitfall of subjects^ social desirability bias 

the desire to be seen in a positive light — that is a shortcoming ^in 

» 

self-report techniques. ^. 
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On the other hand, observation without the addition of self-report, 
restricts the picture to the researcher's point of view. It does not allow 
the subject the opportunity to explain actions in terms of emotional or 
cognitive motivations and effects the researcher can only infer these 
dimensions based on behaviors. Ethnography also presents the problem of 
"co-optation" of the observer (Hall, 1^80). Field site personnel often place 

pressures on the ethnographer to become involved in activities, like 

'J 

consultation, that can bias his/her role as an impartial observer. Since the 

ethnographer cannot be at all places at one time, the process of selecting 

areas upon which to focus attention can also create a biased picture. 

Being observed often eauses subjects a certain amount of discomfort or 

stress, especially at the beginning of the study or during "sensitive" 

♦ 

situations (Hord, 1980). Potentially, the presence o^ an ethnographer can 
consciously or unconsiously alter the normal behavior of the subject. .This 
tendency would probably be more pronounced in discontinuous or short-term data 
collection procedures in which the subject has not become accustomed to the 
observer's presence. Finally, ethnographic observation culminates, in massive 
amounts of descriptive information that must then be reduced for analysis. 
This data reduction process tends to be quite costly in terms of staff time, 
as does the data collection phase of ethnogr^fphy. 

Face-to-Face Interviews 

Face-to-face interviews tend to be a comfortable, easy method of data 
collection for study subjects, as* well as for researchers. Unlike pure 
observation, the interviews give the researcher the opportunity to probe for 
clarifying or explanatory information and for data needed to code 
interventions properly. If face-to-face interviews are done fairly frequently 
(every two weeks or so) they can provide enough information to build a fairly 



complete pitture of the implementation effort. If the interviews occur at the 
study site, they also allow. the researcher* to collect contextual data through 
observation. ^ 

However, unless the interviews take place very frequently, which can be 
very cosily if travel is involved, a certain amount of information is Jikely 
to be lost due to memory limitations. Subjects are most likely to remember 
interventions which have taken place closest to ,the time of the interview 
("recency" effect). There will also be a loss of data on interventions that 
the respondent fails to s6e as important to the research effort. This loss of 
seemingly insignificant data can well be a limitation of any self-report 
method. The tendency of the subjects to present themselves in a positive 
light is also an inherent limitatipn of interviews* and other forms of 
self-r6port. Face-to-face interviews also tend to generate large amounts of 
descriptive data which must then be reduced or summarized in some fashion for 
data analysis. — 

Telephone Interviev/s 

Telephone interviews are characterized by most of ttie same strengths and 
weaknesses as face-to-face interviews. However, sjnce they generally involve^ 
less cost, they can usually take place more frequently. With the increase in 
frequency of contact, comes the ability to capture more of the incidental 
interventions that may be forgotten during longer intervals. Telephone 
interviews als^o tend to be less disruptir^ to school routine and therefore 
more acceptable to research subjects if they are to take place on a regular 
basis. ' * . * 

Telephone interviews without previous face-to-face contact, however, 
engender a certain degree of discomfort on the part of both the interviewer 
and the respondent. Some subjects in the pilot study said that they would not 



like to do telephone interviews with someone they had never seen. It also 
takes ,a longer time to build trust and op.enness within a faceles'i . , 
relationship. On the other hand, with some respondents, the anonymity might 
increase honesty and openness, decreasing the desire to present oneself in a 
purely positive light. The lack of non-verbal cues can, in some cases, limit 
the interpretation of verbal reports. However, this did not appear.,to be a 
significant problem in any of the ^.studies reported here.- 

Tape-Recorded Reports 

Tape-recorded reports of interventions provide the advantage of immediacy^ 
— they allow the subject to report on actions shortly after they occur. If 
used as intended, tape-recorded reports would decrease the loss of information 
due to memory failure. It is also a very inexpensive method of -data 
collection involving little staff time or travel expenditure. Tape-recorded 
reports also tend to include more descriptive information and anecdote than 
written logs, since writing is often viewed as a burden. 

Contrary to expectations, a few of the subjects in the pilot study (Study 
2) f,elt uncomfortable about, or intimidated by, tape-recording. This 
technique therefore resulted in a lower degree of cooperation or compliajice 
with the study procedures in the pilot than did techniques involving 
researcher/ subject contact. "Another disadvantage of this technique is that it 
does not allow the researcher to probe for additional or clarifying 
information. This sometimes results in incomplete or incomprehensible data 
for coding and subsequent quantitative analysis. Lack of non-verbal cues can 

•r 

also be a limitation; but, tape-recording does provide access to tonal quality 
of %f\e voice in interpreting interviews, which ^he final procedure to be 
discussed, written logs, does not. ^ 

_ _ a 
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Written Reports 

Havjng the subject keep written reports or logs is probably the least 
expensive form of data collection on interventions- Like tape recording, it 
also :has the advantage of immediacy; the subject can jot down notes When 
something occurs. This would result in a more detailed, complete picture of 
the change effort,. Written logs also involve considerably less data reduction 
for analysis than do the other meth-ods discussed. ' - * - 

. The main problem with written reports is the sybjects' reluctance to 
comply. WViting- appears to be m^l^emanding, or more of a burden, than the 
other forms of d|*^a co.l lection, even if it does not really "involve the 
- T expendittrf^*5dFmore time. Like tape-recording^ it does not allow for 

researcher probing or cTarif ic-*tion. Also, like th6 tape-recording technique, 
/-^the focus is on the small, incidental events. When the .focus is at this- 
level, one never does get. a sense of the whole picture from the subject's 
point of view. The researcher often must infer or deduce the higher levels of 
interventions, such as strategies and game plans., him/herself. Links between 
interventions which clarify the flow of events are often missing. 

Figure 4 serves as a summary af, the strengths and weaknesses of the . 

various procedures .for intervention documentation just discussed. It would 
appear that some combination of procedures would result in, elimination of 
several of the weaknesses and enhancement of the strengths of any ope 
technique . The CBAM staff found this to be true in the PTI study (Study 3) 
in which most of the procedures discussed were employed in varying degrees. 

Strengths and Limitations of a Hybrid/Multi-Informant Method of Documenting ' 
Interventions; The PTI Study 

The procedures used to document* interventions in the PTI study (Study 3) 

V 

included both face-to-face and telephone interviewing, some ethnographic 
O ' 24 . 
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Figure 4 

Strengths and Weaknesses of Procedures for Documenting Interventions 



Observation 
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Provides contextual 
information 



Face-to-F^ce 
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Comfortable/easy for 
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Clear/codable data set 
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Interviews 
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information due to 
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Tape-Recorded 
Reports 

Immediacy--can capture 
events shortly after 
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Researcher bias 

Co-optation of ob- 
server-loss of 
objectivity 

Discomfort/stress 
to subjects 

Potentially disruptive 
to normal behavior/ 
routine 

Expensive 

Requires much data 
reduction 



Loss, of data due to 
memory limitations 

Loss of seemingly 
insignificant 
incidents 

Subject "social 
desirability" bias 

Requires data 
reduction 



Potentially uncomfor- 
table for subjects & 
'researchers (initially) 

Takes longer to build 
trust/openness 

Lacks non-verbal cues 

Subject "social 
desirability" bias 

Requ i f es^ mut h "data 
reduction 



Uncomfortable for some 
subjects 

Doesn't allow for 
probing by researcher 
(less complete/clear 
data) 

Lacks non-verbal cues 

Focus is at incidental 

J ev-al-r-rmus-t-deduce^ - 
higher levels — — 



Subject reluctance to 
comply 

Focus is at incident 
level --must deduce 
higher levels 

Lacks non-verbal cues 
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observation and some written iiocumentation. Information was gathered from a 
number of informants at each site. D^ta collection also proceeded for one 
full school year. The combined methodologies, as well as the intensity and 
length of the investigation, acted to negate many of the weaknesses of more 
limited, single-method studies. 

The combined procedures resulted in codable data that fairly represented 
the implementation processes at all nine study sites. In the study , 
debriefings, all principals felt that we had gotten a fair representation of 
the actions and events related to the innovation that had occurred at their 
schools. Some principals went so far as to say that we had "gotten it all" or 
98% of all that had happened regarding the innovation that year. They said 



that we might have gotten more details if we had been at the site i^ore 
frequently, but that we would not have gotten a different picture. 

The principals were hinting at the notion of "diminishing returns". If 
*pure ethnographic procedures had been used, it is possible that we would have 
collected more incidental interventions. But the cost in moving from 90% 
complete data to 99% would. have been high, in terms of dollars and staff time. 



,and— [>robabl}/^WGU 14— have "--gotten --4is — very fevA-^^dditional significant - 
interventions. Details and links might have been filled in, bcit the total 
picturfe is not likely to have changed. The techniques employed resulted in a 
picture which included the large plans and events, as well as a flavor fdr the' 

i 

every day occurrences. 

Aljl of ;|the principals reported that they felt comfortable with the 
procedu^res, apd most expressed the feeling that they had enjoyed participating 
because, it had given them the opportunity to reflect on their behaviors. The 



proceduj^es, and the ^study as a whole, increased awareness of what and how much 

! ! 

they were doing to faciltiate -implementation of a new program. The procedures 



ERIC 



\ 
\ 

\ 
\ 



26 o 



were not perceived as a burden or a disruption to routine/ Only one of the 
principals mentjoned a problem, and that had to do with finding space where 
the interviews with teachers might take place. Some of the teachers felt^a 
bit "imposed upon" at the beginning of the study, but once they became more 
familiar with the researchers and with the purpose of the study, this was no 
longer a problem. 

One of the main contributing factors to the success of the techniques 
used in the Principal -Teacher Interaction study was continuity in contact with 
the school. One researcher was assigned to each of the schools. This one 
person was then responsible for all data* col lection, follow-up, and feedback 
involving that site. This enabled the researcher to build a total pi cture of 
what was going on in that school. Two of the principals mentioned that they 
were amazed at how much the researcher remembered about what had happened 
earlier in the year. One principal said he thought the researcher may have 
had a better overall understanding of what was happening than he did. The 
assignment of one liaison to each site also enabled the development of trust 
and rapport between researcher and subjects which probably resulted in 
4^nered5^d-opennes'S--'dnd~more'-eompl-et-e--and-va-14d--da-t-^ — — — 

The inclusion of multiple respondents or informants for each study site 
was a very significant strength of this study. To begin with, having more 
than one informant per site resulted in a larger, more complete data set than 
would have been possible with just one informant. Though the additional 
subjects or informants — the assistant principals, teachers and district 
resource people reported many of the same interventions as the principals, 
they also reported quite a few difiVerent ones. 

Having multiple reports on the same intervention also has advantages. It 
provides an opportunity to assess the degree of bias and accuracy of the data 



obtained by self-report procedures. Basic descriptions of the same action or 
event reported '?)y different individuals tended to be quite similar* The pilot 
study, in which principals and teachers reported on the principal's 
interventions, also revealed a high degree of verification of principal 
self-rep6rts» Often one person can fill in details that the other person has 
forgotten* Different informants can also provide different viewpoints on 
effects of interventions* 

Despite the fact that the hybrid/multi-informant methodology overcomes 
many of the weaknesses of individual methods, a few limitations persist. As 
in any self-report procedure, the tendency to present oneself in the best 
light ("halo effect") 5; till exi sts in t he hybrid p rocedure* This tendency 
reveals itself during intervention coding, such as when trying to determine 
who was the source of an action. Some principals would give the impression 
that they had initiated an action when i,t was actually the "so-called" target 
v/ho was the initiator* It is o-^ten possible to ferret out these biases and 
determine what actually happened when more than one individual reports on the 
action* However, many interventions were only reported by one source* 

ZZ Anather_lini3ia±ioa^o/L±he-4nuJ^^^ the PTI study is its 

dependency on the skills and talents of the researcher* The researcher must 
^|.sess good interpersonal skills, as well as a thorough understanding of the 
research questions and objectives, to obtain complete and trustworthy datai* 
In projects like the PTI study, in which different researchers are responsible 
for each study site, it is necessary to have highly specified procedures, 
intensive researcher training and ongoing communication between researchers in 
order to make vaVid cross-site comparisons* 

FinaTTy j^he^osF s procedures used in the PTI 

study is the intervening effect of the research upon the study subjects' usual 



behavior. Almost all of the principals said that participation in the study 
increased their awareness of their own actions pertaining to the innovation. 
Several of them admitted that their involvement in the study may well have . 
caused them to act differently — to intervene more — than they would have 
otherwise. In the words of one principal, involvement in the research "put a 
little extra push behind me." This increase in innovation-related behavior 
could mean that the PTI Study descriptions of principals as change 
facilitators are somewhat optimistic. A few of the principals felt that study 
participation may have affected their teachers also, causing them to place a 
higher priority on the innovation than they might have otherwise, it is 
possible that a participant observer who has assimilated him/herself in to the 
context might be able to circumvent this intervening effect. On the other 
harld, it also may be impossible to control for this effect with any procedure 
which can be used to document interventions. 

Summary, Conclusions and Implications 

The paper has explored the techniques and accompanying problems of 
xoUecting^ data on -interventions^^r^-^.actions- or,-ev^nts_j^^hji:h-Jaf^^^ 
individuals in their use of an innovation. The focus of the paper is on an 
eclectic methodology, used in the PTI study, which was developed through 
experience gained with procedures employed in earlier studies. Strengths and 
v/eaknesses of procedures employed in these preliminary studies, as well as 
those employed in the PTI study, were discussed. 

One can conclude that a technique which combines several procedures as 
well as several perspectives, as does the technique used in the PTI study, is 
an effective means of capturing the events and actions which influence the 
implementation of an innovation. A methodology which employs several 
self-report techniques and seve^^nnformants can be almost as effective in 
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developing the total picture as ethnographic methodologies which are 
considerably more expensive. The hybrid approach can also circumvent many of< 
the problems associated with more limited self-report methodologies, such as 
the tendency of the subject to present him/herself in a positive light. 

Data collected in this way is highly descriptive, yet focused. This 
allows the researcher to analyze the data both quantitatively (Hord and Hall, 
1982) and qualitatively (Stiegelbauer, Goldstein & Huling, 1982) without an 
inordinate amount of reformulation. The techniques also provided the 
opportunity to collect rich contextual data which will serve to complement the 
intervention data and fill out the total picture of implementation (Griffin & 
Hall, 1982), The three other papers that compose thi s symp osium will describe 
these qualitative and quantitative techniques for analysis of interventions 
and context which are being performed on the PTI Study data (Hord and Hall, 
1982; Stiegelbauer, Goldstein & Huling, 1982; Hall & Griffin, 1982). 

The study of interventions is an important pursuit. In order to 
understand the process of change in schools, one must explore v/hat significant 
individuals do to make change occur. Before one can train people 
sucH ^ school principals, to be nfore effective leaders one must understand 
what effective leaders do to bring about beneficial changes. If one aims to 
determine if certain school programs are effective, it is important to look 
not only at effects and at use of the program, but also at events and actions 
which occur that potentially influence outcomes. To accomplish these tasks it 
is necessary to develop effective and efficient methods for studying what ^ 
people do, the interventions they make, to influence school change efforts. 

The methods employed for the PTI study were both effective and efficient 
in accomplishing the goals of the study. Similar methods could well be 
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employed in future, studies of school change, as well as in other studies Which 
focus upon individual behaviors within organizational settings. 
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